Multiple exposure high dynamic range image capture

ABSTRACT

Techniques for creating a High Dynamic Range (HDR) image within a consumer grade digital camera from a series of images of a scene captured at different exposure levels, and displaying the HDR image on the camera&#39;s built-in display, are provided. The approach employs mixing images of the series to incorporate both scene shadow and highlight details, and the removing of “ghost” image artifacts appearing in the mixed HDR image resulting from movement in the scene over the time the series images are captured. The low computational resource utilization of the present invention&#39;s image mixing and ghost removal processing operations, along with the present invention&#39;s ability to commence image mixing and ghost removal prior to the acquisition of all series images, can significantly reduce the time required to generate and display a tone mapped HDR image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of U.S. patent ApplicationSer. No. 12/763,693, entitled “MULTIPLE EXPOSURE HIGH DYNAMIC RANGEIMAGE CAPTURE”, FILED Apr. 20, 2010; which application claims priorityunder 35 U.S.C. §119(e) to U.S. Provisional Application Ser. No.61/171,936, entitled “HDR from Multiple Exposures” filed on Apr. 23,2009; all of the foregoing applications are incorporated herein byreference in their entiereties.

BACKGROUND OF INVENTION

1. Field of Invention

This invention relates to the acquisition and processing of images thatdisplay the full range of gray shades that appear in a physical scene,often referred to as a “High Dynamic Range” or “HDR” image. Moreparticularly it relates to a system and method for the image capture andprocessing of a HDR image in a digital image capture device such as aconsumer grade digital camera.

2. Discussion of Related Art

Images captured by digital cameras are most commonly Low Dynamic Range(LDR) images, in which each image pixel is comprised of a limited numberof digital bits per color. The number of digital bits per pixel iscalled the digital pixel bit width value. This number is commonly 8bits. Such 8 bit pixels can be used to form an image with 256 differentgray levels for each color at each pixel location. In a LDR image of ascene, shadow areas of the scene are depicted as being completely black(black saturation), bright sunlit areas of the scene are depicted asbeing completely white (white saturation), and scene areas in betweenare shown in a range of gray shades. A High Dynamic Range (HDR) image isone that has digital pixel bit width values of greater than 8 bits, 16bits per pixels is a possible value. In such an image the full range ofgray shades that appear in a physical scene can be displayed. These grayshades provide image details that are present in the scene's shadowregions, highlight regions and mid tone regions that are missing fromthe LDR image. Thus, in an HDR image, scene details are present in imagedark areas that are in shadow due to their proximity next to tailbuildings and beneath trees, in light areas directly illuminated bybright sunlight, as well as in mid-illumination areas that are lightedbetween these 2 extremes.

An HDR image can be captured by acquiring multiple LDR images of a scenethat are captured at different exposure levels. These multiple LDRimages are called a bracketed exposed image series. A low exposure levelwill properly capture the gray shades in scene areas fully illuminatedby bright sunlight and a high exposure level will property capture thegray shades in scene areas completely shielded from the sun and sky bybuildings and trees. However, at the low exposure level the areas of thescene in shadow will be completely black, in black saturation, and showno detail, and the mid-tone areas will lose detail. Further, at the highexposure level, the highlights of the scene will be completely white, inwhite saturation, and show no detail, and the mid-tone areas will againlose detail. Thus, a third, mid exposure level Image, which properlycaptures mid level gray shades, is often acquired as well. By mixingthese three LDR images, an HDR image can be generated that depicts thefull gray scale range of the scene.

Deriving a HDR image from a bracketed exposed image series currentlyrequires a complex implementation that employs an expensivecomputational engine. This is due to the need to perform 3 separateprocessing operations to properly mix the bracketed exposed image seriesinto a single HDR image, and a fourth to convert the resulting image,which is now composed of pixels with digital pixel bit width values ofgreater than 8 bits per color, into one that can be displayed oncommonly available 8 bit per pixel per color displays. These fourprocessing operations are:

“Imago Registration” for accurately aligning the multiple images one toanother;

“Image Mixing” for blending the multiple images together with the properweighting;

“Ghost Removal” for removing location shifted replications of sceneobjects, or ghosts, that would appear in the mixed HDR image, due to themovement of these objects over the time the multiple images wereacquired; and

“Tone Mapping” for preparing the final HDR image for presentation on aconventional displays that are limited to displaying 8 bit per pixel percolor image pixels.

To execute these four processing operations requires the performance ofa large number of floating point operations over a short period of time,as can be seen from a review of “High Dynamic Range Imaging Acquisition,Display, and Image-Based Lighting, authors Erik Reinhard, SumantaPattanaik, Greg Ward and Paul Debevec, published by Morgan KaufmannPublishers, copyright 2005 by Elsevier, inc. This is especially the casefor the image mixing and ghost removal processing operations. Thus,powerful and expensive computational engines (Central Processing Unitsor CPUs) need to be used. Their expense can possibly be tolerated forprofessional digital cameras use, but for inexpensive “Point and Shoot”digital cameras, which incorporate limited processing power CPUs, theyrepresent an impractical solution.

An HDR Image can be created from a bracketed exposed image seriescaptured by an inexpensive digital camera by uploading the image seriesfrom the camera to a general purpose computer, such as Personal Computer(PC). An image processing application, such as Adobe Photoshop, can beused to perform the required complex HDR image combining process on adesktop. This approach is not efficient or convenient and does not meetdemands to reconstruct an HDR image on the camera's built-in displayshortly after its capture.

Thus there exists a need for an in-camera method and apparatus that canrapidly create a HDR image from a bracketed exposed image series, anddisplay it on the camera's built-in display shortly after capture, usinga limited processing power CPU.

SUMMARY OF INVENTION

It is therefore desirable to:

(a) effect a mixing operation on a series of two or more images of ascene, such series images having been registered one to another, eachimage composed of pixels containing digital bits, to generate acomposite image in which each pixel contains a number of digital bits,the number being greater than the number of digital bits contained inany series image pixel, in a processing operation resource efficientmanner; and

(b) effect a ghost removal operation that removes location shiftedreplications of scene objects appearing in mixed image data, the mixedimage data generated by a digital image mixing process applied to sceneimages acquired at different exposure levels and times, in a processingoperation resource efficient manner.

According to a first aspect of the present invention, a registered,captured bracketed exposed image series composed of two or more LDRimages is mixed to generate a HDR image with digital pixel bit widthvalues greater than that contained in any of the initial LDR imagepixels. Series images at different exposure levels are captured, where aseries image exposed at a first exposure level is exposed less than aseries image exposed at a second exposure level, which is exposed lessthan a series image exposed at a third exposure level, which is exposedless than a series image exposed at a n^(th) exposure level. Anormalized image exposure level for each image in the series is derivedby using the exposure level of the least exposed image of the series asthe reference exposure level, and is employed in an image mixingprocess, wherein series images are blended together, two at a time. Theimage captured at the lowest exposure level of the series is firstblended together with the image captured at the next highest exposurelevel in the series, to generate a first intermediate image. Thegenerated first intermediate image is then blended together with theimage captured at the next highest exposure level of the series togenerate a second intermediate image. If the bracketed exposed imageseries is composed of two images, the mixing process stops at thegeneration of the first intermediate image, and the generated HDR imageoutput is the first intermediate image. If there are three images in theseries, the generated HDR image output is the second intermediate image.If there are more images in the bracketed exposed image series thanthree, each generated intermediate image is blended with the imagecaptured at the next highest exposure level of the series to generate anext intermediate image, until there are no more series images left toblend. In this case, the HDR image output is the image generated by theblending of the last remaining image in the series with the previouslygenerated intermediate image.

This mixing process operation greatly reduces the processing powerrequired to mix a bracketed exposed image series to generate a HDRimage, while minimizing processing latency. The act of normalizing imageexposure level for each captured image to the exposure level of thelowest exposed (darkest) image of the bracketed exposed image seriesallows the mathematical operations employed by the blending processes tobe mostly restricted to summations and multiplications, thus avoidingthe use of division operations that have high computational resourcerequirements.

According to a second aspect of the present invention, a two stagecomputational resource efficient process is used to remove locationshifted replications of scene objects, or ghosts, appearing in the mixedHDR image data generated by a digital image mixing process applied to aseries of scene images acquired at different exposure levels and times.The second stags process removes ghosts that remain after the executionof the first stage of the ghost removal process. In the first stage thevariance of the luma of every pixel of the HDR image as compared to thepixels of a reference image is calculated, pixels in HDR image regionswith variances exceeding a first threshold are replaced with pixels fromthe corresponding image regions of one the image series images, which isused as the reference image. This procedure removes a major part of theghosts, but some ghosts may still remain. In the second processingstage, the luma of pixels of the ghost reduced HDR image, the firstprocessed HDR image, resulting from the first stage ghost removalprocessing operation, are compared with the luma of pixels from thecorresponding image regions of same image series reference image usedfor the first ghost removal stage. Ghost residuals are detected bycalculating the differences between the luma of the first processed HDRimage pixels and the luma of the pixels of the reference image. A secondthreshold based on the peak of these differences is generated. Pixels inHDR image regions exceeding this second threshold are replaced withpixels from the corresponding image regions of the reference image toproduce a second processed HDR image.

In accordance with a third aspect of the present invention, theinvention is incorporated within a digital camera that captures a seriesof two or more digital images of a scene at different exposure levels,and at different times, to generate a tone mapped HDR image, that can bedisplayed shortly after the images of the series are captured, on thecamera's built-in display. Such a digital camera includes the imagemixing and ghost removal processing operations of the present invention,with image registration, tone mapping, general digital camera imageprocessing, control, image sensing and display technologies that arewell known in the art.

By mixing two images of the series at a time, and removing ghosts fromthe mixed HDR image with respect to an image of the image series, themixing and ghost removal processing operations of the present inventioncan commence prior to the capture of all the images that comprise theimage series. In some cases image mixing can commence immediately afterthe capture of the second image of the series. The low computationalresource utilization of the present invention's image mixing and ghostremoval processing operations, along with the present invention'sability to commence image mixing and ghost removal prior to theacquisition of all series images, can significantly reduce the timerequired for a digital camera with low processing power to generate atone mapped HDR image and display the HDR image on its built-in display.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. In thedrawings, each identical or nearly identical component that isillustrated in various figures is represented by a like numeral. Forpurposes of clarity, not every component may be labeled in everydrawing, in the drawings:

FIG. 1 is a block diagram of a digital camera or other image captureapparatus that captures a plurality of digital images of a scene, atdifferent exposure levels and at different times, and displays theseimages on the camera's built-in image display;

FIG. 2 is a high level block diagram of an embodiment of the presentinvention illustrating processing modules as implemented in a digitalcamera;

FIG. 3 is a block diagram of an embodiment of the 2 Image BlendingEngine of the present invention;

FIG. 4 is a flow chart illustrating the complete image mixing processsequence of an Image Mixer processing method of the present invention;

FIG. 4A details the process used by the 2 Image Blending Engine of FIG.3;

FIG. 5 is a block diagram of an embodiment of the Ghost Removerprocessing module of the present invention;

FIG. 6 is a flow chart illustrating the processing sequence of a GhostRemover processing method of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention will now be described more fully hereinafter withreference to the accompanying drawings, which form a part thereof, andwhich show, by way of illustration, a specific embodiment by which theinvention may be practiced. The invention may, however, be embodied inmany different forms and should not be construed as limited to theembodiment set forth herein; rather, this embodiment is provided so thatthis disclosure will be thorough and complete, and will fully convey thescope of the invention to those skilled in the art. Among other things,the present invention may be embodied as methods or devices.Accordingly, the present invention may take the form of an entirelyhardware embodiment in the form of modules or circuits, and entirelysoftware embodiment in the form of software executed on a generalpurpose microprocessor, an application specific microprocessorprocessor, a general purpose digital signal processor or an applicationspecific digital signal processor, or an embodiment combining softwareand hardware aspects. Thus, in the following description, the terms“circuit” and “module” will be used interchangeably to indicate aprocessing element that executes an operation on a input signal andprovides an output signal therefrom regardless of the hardware orsoftware form of its implementation. Likewise, the terms “register”,“registration”, “align” and “alignment” will be used interchangeably toindicate the process of causing like objects to correspond one toanother, and be in correct adjustment, regardless if the mechanismemployed to bring about such correspondence is implemented in the formof hardware or software. The following detailed description is,therefore, not to be taken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may. As usedherein, the term “or” is an inclusive “or” operator, and is equivalentto the term “and/or”, unless the context clearly dictates otherwise. Theterm “based on” is not exclusive and allows for being based onadditional feelers not described, unless the context clearly dictatesotherwise. In addition, throughout the specification, the meaning of“a”, “an”, “and” and “the” include plural references. The meaning of“in” includes “in” and “on”. Also, the use of “including”, “comprising”,“having”, “containing”, “involving”, and variations thereof herein, ismeant to encompass the items listed thereafter and equivalents thereofas well as additional items.

FIG. 1 shows a digital camera or other image capture apparatus whichincludes an Imaging Optical System 105, an Electronically ControlledShutter 180, an Electronically Controlled Lens iris 185, an OpticalImage Sensor 110, an Analog Amplifier 115, an Analog to Digitalconverter 120, an Image Data Signal Processor 125, an Image Data StorageUnit 130, an Image Display 106, and Camera Controller 185. The ImageData Storage unit could be a memory card or an internal nonvolatilememory. Data of images captured by the camera may be stored on the ImageData Storage Unit 130. In this exemplary embodiment, it may also includeinternal volatile memory for temporary image data storage andintermediate image processing results. This volatile memory can bedistributed among the individual image data processing circuits and neednot be architecturally located in a single image data storage unit suchas Image Data Storage Unit 130. The Optical System 105 can be a singlelens, as shown, but will normally be a set of lenses. An Image 190 of aScene 100 is formed in visible optical radiation onto a two-dimensionalsurface of an image sensor 110. An electrical output 195 of the sensorcarries an analog signal resulting from scanning individualphoto-detectors of the surface of the Sensor 110 onto which the Image190 is projected. Signals proportional to the intensity of lightstriking the individual photo-detectors are obtained in the output 195.The analog signal 195 is applied through an Amplifier 115 to an Analogto Digital Converter 120 by means of amplifier output 102. Analog toDigital converter 120 generates an image data signal from the analogsignal at its input and, through output 155, applies it to Image DataSignal Processor 125. The photo-detectors of the Sensor 110 typicallydefect the intensity of the light striking each photo detector elementin one of two or more individual color components. Early detectorsdetected only two separate color of the image. Detection of threeprimary colors, such as red, green and blue (RGB) components, is nowcommon. Currently image sensors that defect more than three colorcomponents are becoming available.

Multiple processing operations are performed on the image data signalfrom Analog to Digital Converter 120 by Image Data Signal Processor 125.Processing of the image data signal, in this embodiment, is shown inFIG. 1 as being effected by multiple image data signal processingcircuits within Image Data Signal Processor 125. However, these circuitscan be implemented by a single integrated circuit image data signalprocessor chip that may include a general purpose processor thatexecutes algorithmic operations defined by stored firmware, multiplegeneral purposed processors that execute algorithmic operations definedby stored firmware, or dedicated processing logic circuits as shown.Additionally, these operations may be implemented by several integratedcircuit chips connected together, but a single chip is preferred. FIG. 1depicts the use of image data signal processing circuits 135, 140, 146,and 150 connected in series to effect multiple algorithmic processingoperations on the image data signal from Analog to Digital Converter120. The result of these operations is a stored nonvolatile digitalimage data that can be viewed either on the digital camera's internalImage Display 106 of FIG. 1, or an external display device. This viewingcan be effected either by the physical removal of a memory card from thedigital camera and the reinsertion of this card into an external displaydevice, or the electronic communication of the digital camera with anexternal display device by the use of a Universal Serial Bus (USB)connection, or a Wi-Fi or Bluetooth wireless local area network.

Additional processing circuits as indicated by the dots 175 betweencircuit 146 and 150, can be included in the digital camera's image datasignal processor. The series structure of the image data signalprocessor of the present embodiment is known as a “pipe line”architecture. This architectural configuration is employed as theexemplary embodiment of the present invention, however otherarchitectures can be used. For example, an image data signal processorwith a “parallel architecture”, in which one or more image data signalprocessing circuits are arranged to receive processed image data signalsfrom a plurality of image data signal processing circuits, rather thanafter they have been processed serially by all preceding image datasignal processing circuits, can be employed. A combination of a partialparallel and a partial pipeline architectures is also a possibility.

The series of image data signal processing circuits of Image DataProcessor 125 is called an “image processing pipe”. The presentinvention adds image data signal processing circuits shown in FIG. 2 tothose routinely included in the image processing pipe of a digitalcamera. Image data signal processing circuits routinely included in theimage processing pipe of a digital camera include circuits for WhiteBalance Correction (WBC), Lens Shading Correction (LSC), GammaCorrection (GC), Color Transformations (CTM), Dynamic Range Compression(DRC), Demosaicing, Noise Reduction (NR), Edge Enhancement (EE),Scaling, and Lens Distortion Correction (LDC). As depicted in FIG. 2,the present invention adds an Image Registration Processor (IRP) circuit210, an Image Mixer (M) circuit 220, Ghost Remover (GR) circuit 230, andTone Mapping Processor (TMP) circuit 235 to the complement of image datasignal processing circuits discussed above. Image Storage 200 of FIG. 2stores the digital data of a series of two or more images of a scene,each series image composed of pixels containing digital bits, thesedigital bits having been processed by image data signal processingcircuits in the image processing pipe. Image Storage 200 could sharememory with Image Storage 130 of FIG. 1, however, memory resources usedfor temporary image data storage and intermediate image processingresults, or totally separate volatile or nonvolatile memory resources,could provide the memory resources used by Image Storage 200.

Referring to FIG. 1, Camera Controller 165, through line 145 andControl/Status lines 160, causes Electronic Shutter 180, Electronic iris185, Image Sensor 110, Analog Amplifier 115, and Analog to Digitalconverter 120 to capture and convert to digital image data a series ofimages of a scene. These images are captured at different exposurelevels, processed by image data signal processing circuits, and storedin Image Storage 200 of FIG. 2. Image Registration Processor 210 readsthe image series digital image data stored in Image Storage 200 andregisters counterpart pixels of each image of the image series one to tothe other. Image registration is executed before image mixing in orderto pixel to pixel align all series images. Due to both camera and objectmovement occurring during image series capture, such alignment isnecessary for image mixer 220 of FIG. 2 to be able to properly combineseries image pixels and form an image with the captured scene's fullrange of gray shades. Such an image is often referred to as a “HighDynamic Range” or “HDR” image. In Image Mixer 220, each series imagepixel of each captured series image is combined with its pixelcounterpart in each captured series image. Thus, an image pixelrepresenting a particular position on the edge or within the body of anobject appearing in a first series image is mixed with its counterpartlocated at the same position on the edge or within the body of the sameobject appearing in a second series image. In this regard, the locationof a pixel in an image is with respect to the object in which it is partof, not to the fixed coordinate system of the defined by the verticaland horizontal outer edges of the image.

Image Registration Processor 210, in general, employs a first imagecaptured at a nominal exposure setting of the camera as a referenceimage to which all the other images of the series are aligned. A numberof techniques are in current use for image alignment and registration. Agood example is described in “High Dynamic Range Video”, S. B. Kang, M.Uyttendaele, S. Winder, and R. Szeilski, Interactive Visual Media Group,Microsoft Research, Redmond, Wash., 2003. The approach described handlesboth camera movement and object movement in a scene. For each pixel amotion vector is computed between successive series images. This motionvector is then refined with additional techniques, such as hierarchicalhomography, to handle degenerate cases. Once the motion of each eachpixel is determined, frames can be warped and registered with the chosenreference image. The images can then be mixed by Image Mixer 220 into anHDR image.

The Image Mixing Process

Image Mixer 220 has the ability to mix an unlimited number of images,but employs an image blending engine that mixes the pixels of 2 imagesat a lime. The 2 Image Blending Engine of Mixer 220 is shown in FIG. 3.In the preferred embodiment of the present invention, the blendingengine blends the pixels of a first 8 bit image whose digital image dataappears on input line 300 and the pixels of a second 8 bit image whosedigital image data appears on input line 305. Images with bit widthsthat are wider than 8 bits, for example 10 bits, or narrower, forexample 7 bits can be used. The Flow Chart of FIG. 4 illustrates thecomplete image mixing process of Image Mixer 220, and FIG. 4A detailsBlock 420 of FIG. 4, the process used by the 2 Image Blending Engine ofFIG. 3.

Referring to FIG. 3, the image mixing process of the present inventionblends 2 images during each image mixing operation. The first 2 imagesto be blended are both taken from the captured image series, whereineach image of the captured image series has been previously registeredwith a reference image captured at a nominal exposure setting of thecamera. For the initial image mixing operation the series image with alower exposure level provides the First Image digital image data inputappearing on line 300 of FIG. 3, and the series image with a higherexposure level provides the Second Image digital image data inputappearing on line 305. For a second and all subsequent image mixingoperations, a subsequent image of the series is blended with the resultobtained from a previous image mixing operation. For these follow-onmixing operations, the digital image data of a subsequent image of theseries serves as the Second Image digital image data input appearing on305, and the mixed image digital image data result serves as the FirstImage digital image data input appearing on line 300 of FIG. 3. In allcases, the subsequent image of the series has been exposed at a higherexposure level than its immediate image series predecessor.

The Second Image digital image data on line 305 is initially processedin 2 ways. (1) Luma Conversion Circuit 320 extracts the Luma, the blackand white component, of the combined Red, Green and Blue (RGB) componentdata comprising Second Image digital image data 305, and outputs theLuma component of each image data pixel on line 325. (2) ImageNormalizer 310 normalizes the exposure level of each RGB component ofthe Second Image image data on line 305 to the exposure level of areference image, and outputs on line 302 each image data pixel, for eachcolor component, normalized to the reference image exposure level. Notethat the reference image used is not necessarily the same referenceimage used for the registration process previously described. For thepreferred embodiment of the present invention the exposure level of thedarkest image of the series, that is the image which is least exposed,serves as the reference exposure level and all other images of theseries are normalized to it. For example, if the captured image seriesis composed of 3 images, a dark image exposed for 1/64 sec, a mediumimage exposure at 1/18 sec and a bright image exposed at ½ sec, thenormalized value of each pixel of the medium image appearing on line 302would be:Medium Pixel Value_(Normalized)=Medium Pixel Value_(Input)/( 1/16/(1/64))=a Medium Pixel Value_(input)/4;  (1)and

the normalized value of each pixel of the bright image appearing on line302 would be:Bright Pixel Value_(Normalized)=Bright Pixel Value_(Input)/((½)/(1/64))=Bright Pixel Value_(Input)/32  (2)

Therefore, for the preferred embodiment of the present invention:Exposure Level_(Normalized)=Exposure Level_(Series Image)/ExposureLevel_(Least Exposed Series Image) and;  (3)

the normalized value of each pixel of the Second Image Data input online 305 and output on line 302 is:2nd Image Pixel Value_(Normalized)=2nd Image Pixel Value_(Input)/2ndImage Exposure Level_(Normalized)  (4)

The luma component of each Second Image digital image data pixelappearing on line 325 of FIG. 3 is input to Look-up Table (LUT) 315 toobtain a per pixel weighting parameter, W_(i), on lines 330 and 335. Theluma component of each Second Image pixel serves as an index into LUT315, and causes a weighting parameter value, W_(i) between the numberszero and Unity to be output to lines 330 and 335 for each input lumavalue. This value is output in the form of a two dimensional matrixwhere:W _((m,n))=255−Luma_((m,n):)  (6)

Luma_((m,n)) is the Luma component of each Second Image digital datapixel at image coordinates (m,n), which, for the preferred embodiment ofthe present invention, can attain a maximum value of 255, since theembodiment blends the pixels of 8 bit series images, and 255=Unity,which, represents Table 315's 100% output-value, because the lumacomponent Second Image pixel values serving as indexes into LUT 315, are8-bit digital values with a number range from 0 to 255. Therefore,defining 255 as Unity allows for a direct mapping from input index valueto output weighting parameter value and reduces weighting parameterapplication computational work load. Other values of Unity can bechosen. For example, if the luma component Second Image pixel valuesserving as indexes info LUT 315 are 10-bit digital values, with a numberrange from 0 to 1024, it would be appropriate and beneficial to assignUnity the value of 1024.

For the preferred embodiment, the weighting parameter value output fromLUT 315 linearly decreases as the luma component Second Image pixelvalue, serving as the index into LUT 315, increases. Other LUTfunctions, for example, trapezoidal shaped functions in which theweighting parameter value obtained from LUT 315 remains at apredetermined value and starts to linearly increase when the lumacomponent Second Image pixel value index decreases below a threshold,may be used as well. The choice of LUT 315 functions is based on theobservation that when two images are mixed, one which is highlysaturated, due to being exposed at a high exposure level, perhaps at along exposure time, and the other dark, due to being exposed at a lowerexposure level, perhaps at a short exposure time, it is desirable toapply a low weight to the highly saturated pixels of the image with thehigh exposure level, while applying a high weight to the counterpartpixels of image with the low exposure level. This will result in a mixedimage with fewer highly saturated pixels, since many have beensupplanted by counterpart, properly exposed pixels. The result is amixed image with greater detail in its highlight areas.

The present invention is not restricted to the use of a single LUT 315.A plurality of LUTs can be used. In this case a different LUT can beassociated with each Series Image for obtaining the weighting value, ortwo or more Series Images can be associated with the same LUT from aplurality of provided LUTs. These LUTs can, for example, be populatedwith weighting parameter values responsive to Series Image exposurelevels.

During the blending operation of the present invention, the weightingparameter is applied, on a pixel by pixel basis, to each color componentof the normalized Second Image digital image data appearing on line 302,and 1 minus the weighting parameter (1−W_(i)) is applied to each colorcomponent of the First Image digital image data appearing on line 300.The pixel by pixel blending operation of the present invention isdefined by the following equation:Blended Image Data Pixel=(1−W_(i))×(1st Image DatePixel)+W_(i)×(Normalized 2nd Image Data Pixel)  (5)The processing blocks of FIG. 3 execute equation (5) as follows: TheLuma of the Second Image Data on Line 305 is derived by Luma Conversioncircuit 320 and used by LUT 315 to generate weighting parameter W_(i) onLines 330 and 335. Multiplier 307 multiplies normalized Second ImageDigital Image Data, normalized by Image Normalizer 310, by W_(i) on Line330 and outputs the result on Line 355. W_(i) is also applied to DataSubtractor 340 through Line 335, which outputs (1−W_(i)) on Line 345.First Image Digital Image Data on Line 300 is multiplied by (1−W_(i)) onLine 345 by Multiplier 350 and outputs the result on Line 365. TheNormalized and weighted Second Image Digital Image Data on Line 355 isadded to weighted First Image Digital Image Data on line 365 by adder380. Adder 360 outputs Blended Image Pixels on Line 370. These pixelsare stored in Image Storage 375 and output as 2 Image Blended Image Dataon Line 380.

The pixel blending process used by the 2 Image Blending Engine of FIG. 3is depicted in processing Block 420 of FIG. 4A. The data of the FirstImage to be blended enters the process at 423 and the data of the SecondImage to blended enters the process at 413. The pixel blending processbegins at 445. At 427 a pixel from the Second Image is selected. Theselected Second Image pixel is normalized at 429 and its luma is derivedat 450. The Second Image pixel's luma is used to obtain weightingparameter W_(i), 486, from a LUT at 465. The normalized Second Imagepixel is multiplied by weighting parameter W_(i), 485, at 455. Thenormalized and weighted Second Image pixel enters an adding process at475. The First Image Pixel counterpart of the selected Second ImagePixel is selected at 460 and multiplied by (1−W_(i)) at 470. Theweighted First Image pixel enters the adding process at 475 and isblended with the normalized and weighted Second Image pixel. If thereare more image pixels left to blend, as determined at decision point 480and indicated by “No” on 431, then the next pixel blend cycle isinitiated at 445, causing a next First Image pixel and a next SecondImage pixel to be selected at 480 and 427, respectively and blended asdescribed.

If there are no more image pixels left to blend, as determined atdecision point 480 and Indicated by “Yes” on 433, but there are moreimages in the captured image series to mix, as determined at decisionpoint 495 and indicated by “No” on 487, then another Second Image isselected from the remaining, unmixed series images, whose data willserve as the next Second Image data to be blended. The Second Imageselected will have a higher exposure level than the previous SecondImage selection. Such selection is made by Image Selection Process 425in response to the “No” on 487. Additionally, the 2 Image Mixed ImageOutput Data at 493 is selected by First Image data selection process 435as the next First Image data to be blended, due to decision point 430signaling “Yes” to process 435 at 441, in response to 2 Image MixedImage data begin available at 493. IF 2 Image Mixed Image data is notavailable at 483, as would be the case at the beginning of an imagemixing process, decision point 430 would signal processing Block 440, byplacing a “No” at 437, to select a First Image from the captured imageseries to be blended with a lower exposure level than the Second Imageto be blended selected from the captured image series by processingBlock 425. In this case, Selected Second Image exposure levelinformation is communicated to selection Block 440 at 407, least exposedSeries Image exposure level information is communicated to selectionBlock 440 at 443, and selected Series Image data is communicated toprocessing Block 440 at 439. As depicted in the Flow Chart of FIG. 4,and used in the preferred embodiment of the invention, the Series Imagewith the least exposure level may be selected as First Image.

If there are no more images in the captured image series to mix, theprocess concludes with the Mixed HDR Image Output appearing at 497.

The Flow Chart of FIG. 4 illustrates the complete image mixing processof Image Mixer 220. Block 420, depicting the 2 Image Image Blendingprocess used by the 2 Image Blending Engine of FIG. 3, is highlighted inFIG. 4 and illustrated in detail in FIG. 4A. FIG. 4 also includes theprocessing that precede 2 Image Blending processing Block 420. Thisprocessing is comprised of the capture of a series of images, each imagebeing exposed at a different exposure level, at processing Block 400,the registration of counterpart series image pixels one to another, atprocessing Block 405, the determination of the series image that isleast exposed, at Processing Block 410, and the calculation of anormalized exposure level according to Equation (3) previouslydescribed, referenced to the least exposed image of the series, for eachimage in the series, at processing Block 415. This calculated normalizedexposure level is used by 429 of processing Block 420 to normalize eachSecond Image pixel, as previously described by Equation (4), beforebeing multiplied by a weighting parameter and blended with a weightedFirst Image pixel according to Equation (5) previously described.

The image mixing process of Image Mixer 220 uses only summation andmultiplications, thereby avoiding computationally intensive divisionoperations, and allowing it to be implemented by a fixed pointcomputational engine. In addition, since the mixing approach employed isbased on successive 2 image mixing operations, there is no need to waituntil all series images have been captured before initiating a mixingoperation. The mixing process can begin immediately after just 2 imagesof the series have been captured. These characteristics allow the mixingprocess of the present invention to rapidly create a HDR image from abracketed exposed image series using a limited processing powercomputational engine.

The Ghost Removal Process

Ghost Remover 230 removes location shifted, replications of sceneobjects, or ghosts, that appear in the Mixed HDR Image Output data at497 of FIG. 4, due to the movement of these objects over the time theimages of the series are captured. Fundamentally, ghosts appear due tothe mixing of Series Images in which an object visualized in a FirstSeries Image has moved with respect to its First Series Image locationcoordinates, as visualized in a Second Series Image. Consequently, inthe mixed image, the object may appear in multiple locations with thelocations depending on the object's motion rate and motion direction.The present invention employs a unique 2 stage approach to mitigatingghost images.

Given a mixed image, HDR (i,j), generated from a weighted sum of imagesfrom a captured aligned image series of K images, the present invention,in a first stage of processing, first calculates the variance of theluma value of every pixel of HDR(i,j) as follows:V(i,j)=Σ_(k) W(i,j,k)×P²(i,j,k)−HDR(i,j)²  (7)

Where;

-   V(i,J)=The variance of the luma value of Mixed HDR image pixel,    HDR(i,j), located at image coordinates (i,j) with respect to the    value of Kth Series Image pixel located at image coordinates (i,j),    over the K aligned images of the captured image series;-   HDR(i,j)=The luma value of the Mixed HDR image pixel located at    image coordinates (i,j);-   W(i,j,k)=A normalizing weight applied to the level of the Kth Series    Image pixel located at image coordinates (i,j) to normalize the    Series Image pixel level range to the pixel level range of the Mixed    HDR image; and-   P(i,j,k)=The value of Kth Series Image pixel located at image    coordinates (i,j);    and then replaces a Mixed HDR Image pixel whose luma variance    exceeds a first threshold level with its counterpart pixel from an    aligned reference image, HDR_(ref), the reference image begin chosen    from the captured image series, to generate first processed mixed    image data, HDR_(1st processed).

The first stage of ghost removing processing, described above, is basedon the observation that if no local motion exists in the Series Imagesmixed, the variance of a pixel in the Mixed HDR Image output over the Kaligned images of the captured image series, as defined by Equation (7)above, will be low. The only significant error associated with thisassumption is alignment error, which is around −15 decibels relative tothe variance of each of the K aligned images of the captured imageseries. Since alignment is inherently a global process, it cannotcompensate for local image object local motion, and thus local objectmotion is manifested as high amplitude pixel variance regions. Byanalyzing the high amplitude pixel variance regions in the 2-dimensionalvariance data generated by Equation (7), regions of local motion in theMixed HDR Image output data can be defined, and Mixed HDR Image outputdata pixels with variances above a predefined threshold can be replacedwith counterpart pixels from the reference image. The reference imagechosen is often the least exposed Series Image, but may be a SeriesImage exposed at a higher exposure level.

The present invention's first stage of ghost removal processinggenerates first processed mixed image data, HDR_(1st processed), withless ghosting. However some residual ghosting still remains. A secondstage of processing improves these results by comparing the content ofHDR_(1st processes) with HDR_(ref). In this second stage, ghostresiduals are detected by analyzing the pixel to pixel result obtainedby subtracting the luma of HDR_(ref) from the luma ofHDR_(1st processed). A second threshold level based on the maximum valueof the differences between the luma of HDR_(1st processed) and the lumaof HDR_(ref) is generated. Each HDR_(1st processed) data pixel exceedingthe second threshold level is replaced with its counterpart. HDR_(ref)data pixel, resulting in second processed mixed image data,HDR_(2nd processed), with fewer ghosts. The procedure used by thepreferred embodiment of the present invention's second stage ofprocessing can be summarized as follows:

(a) Generate D₀ = ABS(Luma(HDR_(1st processed)) − Luma(HDR_(ref))); (b)Determine Threshold_(2nd) = Max(D₀) = D_(M0). (c) Replace eachHDR_(1st processed) data pixel exceeding Threshold_(2nd) with itscounterpart HDR_(ref) data pixel, resulting in HDR_(2nd processed) mixedimage data (d) Compare HDR_(2nd processed) with HDR_(ref) and generateMax(D₁) = D_(m1) where D_(m1) = Max((ABS(Luma(HDR_(2nd processed)) −Luma(HDR_(ref)))) (d) If D_(m1) > 60% of D_(M0), Threshold_(2nd) is toolarge and HDR_(2nd processed) may look too much like HDR_(ref). Then:(e)   Segment D_(M0) into 2 levels, where D_(M00) = a value < 0.5D_(M0),and D_(M01) = a value >   0.5D_(M0) (f)   Determine, in percent, theamount of HDR_(2nd processed) image area relative to the full   imagearea, exceeding D_(M01), SIZE_1 (g)   Determine, in percent, the amountof HDR_(1st processed) image area relative to the full   image area,exceeding D_(M00), SIZE_0, where SIZE_1 should be >= SIZE_0. (g)  Calculated SlZE-RATIO = SIZE_1 / SIZE_0 (h)   IF (SIZE_0 > 40% ||(SIZE_RATIO > 2 && SIZE_1 > 8 %)) (i)     Capture Series Images again(j)   Else (k)     Replace pixels of HDR_(2nd processed) image areasexceeding D_(M01), with their     counterpart HDR_(ref) pixels,resulting in HDR_(2nd processed) mixed image data (10) End

The ghost removal process of the present invention can be applied to anyaligned, captured bracketed exposed series of two or more images. Inaddition, two or more HDR_(ref) Images can be employed by the process.When the process is applied to a series of three images, for example,the exposure level of a first series image being lower than the exposurelevel of a second series image and the exposure level of the secondseries image being lower than the exposure level of a third seriesimage, the areas of the regions of the second series image thatcorrespond to mixed image data with variances that exceed the firstthreshold can be used to select between 2 reference images, HDR_(ref1)and HDR_(ref2). In this example, second series image region areas thatare saturated are summed and a ratio of the sum of saturated area to thetotal area of the second series image is used to select HDR_(ref) forthe remainder of ghost removal processing. If the ratio is less than orequal to 0.03 to 1 then HDR_(ref2)=the second series image is selected.If the ratio is greater than 0.03 to 1 then then HDR_(ref1)=the firstseries image is selected. Further, the above selection approach, or onessimilar in nature that are responsive to other image characteristics,such as, for example, the size of image areas with object movement abovea predetermined threshold, or spacial frequency details above apredetermined threshold, can be used to select a HDR_(ref1) for thefirst stage of ghost removal processing and, additionally a differentHDR_(ref2) for the second stage of ghost removal processing.

FIG. 5 is a block diagram of an embodiment of the present invention'sGhost Remover processing module, 230 of FIG. 2. Mixed HDR Image pixeldata is input to Luma Conversion Circuit 515 and First Pixel ReplacementCircuit 540 on line 505 of FIG. 5. Luma Conversion Circuit 515 convertsMixed HDR Image Pixel Data to Mixed Image Luma Pixel Data and, throughline 525, inputs Mixed Image Luma Pixel Data to Variance CalculationCircuit 550. Although not shown in FIG. 2, aligned images of thecaptured bracketed exposed series of two or more images are input to theGhost Remover module 230 on line 500 of FIG. 5, which is connected toReference Image Selection Circuit 510. In the preferred embodiment ofthe present invention, Reference Image Selection Circuit 510 selects theleast exposed series image as the reference image, HDR_(ref), however aSeries Image exposed at a higher exposure level could be selected.Through line 520, HDR_(ref) Pixel Data is also applied to VarianceCalculation Circuit 550. Additionally, line 520 applies HDR_(ref) PixelData to 1st Pixel Replacement Circuit 540, 2nd Pixel Replacement Circuit585 and Luma Conversion Circuit 560. From the HDR_(ref) Pixel Data online 520 and Mixed Image Luma Pixel Data on line 525, VarianceCalculation Circuit 550 generates output Mixed Image Luma Pixel VarianceData on line 530. This Luma Pixel Variance Data is applied to FirstPixel Replacement Circuit 540 through line 530. On line 535, a 1stThreshold Level is also applied to 1st Pixel Replacement Circuit 540.From these inputs, 1st Pixel Replacement Circuit 540 replaces pixels ofthe Mixed Image Pixel Data on line 505, whose Luma variance exceeds thefirst threshold level on line 538, with counterpart pixels from theHDR_(ref) Data on line 520, to generate first processed mixed imagepixel data, HDR_(1st processed), on line 545, which is the output of afirst stage of processing.

The first stage of processing output, HDR_(1st processed), on line 545,is converted to HDR_(1st processed Luma) Pixel Data by Luma ConversionCircuit 585. The output of Circuit 565 appears on line 595 and isconnected to Comparison Circuit 575. HDR_(1st processed) on line 545 isalso applied to 2nd Pixel Replacement Circuit 585. Line 520 appliesHDR_(ref) Pixel Data to Luma Conversion Circuit 500 and 2nd PixelReplacement Circuit 585. Pixel Data to Luma Conversion Circuit 560converts HDR_(ref) Pixel Data to HDR_(ref Luma) Pixel Data, and providesthe HDR_(ref Luma) Pixel Data to Comparison Circuit 575 over Line 570.Comparison Circuit 575 calculates the difference between the eachHDR_(1st process Luma) Data Pixel and its counterpart HDR_(ref Luma)Data Pixel and generates a 2nd Threshold Level based on the maximumvalue of the differences. This 2nd Threshold Level is applied to 2ndPixel Replacement Circuit 585 over Line 580. 2nd Pixel ReplacementCircuit 585 replaces each HDR_(1st processed) data pixel on line 545exceeding the 2nd threshold level with its counterpart HDR_(ref) datapixel on line 520, the resulting 2nd processed mixed image data,HDR_(2nd processed), on line 590, being the ghost reduced output of asecond stage of processing.

The 2 stage ghost removal process used by the Ghost Remover processingmodule of FIG. 5 is depicted in the flowchart of FIG. 6. At Block 600, abracketed exposed series of two or more images, each image exposed at adifferent exposure level and at a different time, is captured. At Block605 these images are registered to each other such that counterpartseries image pixels correspond one another. At Block 610 a referenceimage is selected from the acquired scene images and its pixel data ispassed onto processing Blocks 625, 640, 660 and 695 over processing path615. The reference image chosen is often the least exposed Series Image,but may be a Series Image exposed at a higher exposure level. It doesnot have to be the same reference image as used by the presentinvention's Image Registration Processor 210, which, in general, employsa first image captured at a nominal exposure setting of the camera as areference image to which all the other images of the series are aligned.The Image Mixer of the present invention previously described, whoseimage mixing process is depicted in the flow chart of FIG. 4, performsthe processing at Block 620, with the Mixed HDR Image Output Image DataPixels of FIG. 4 entering processing Blocks 650 and 640. At Block 650the Luma of Mixed Data Pixels is generated and passed onto Block 625,where the variance of each Mixed Image Data Pixel Luma from Block 650,as compared with its counterpart Reference Image Data Pixel from Block610, is calculated. This variance is passed to processing Block 640 andused by Block 640, along with a 1st Threshold which enters processingBlock 640 along path 635, Mixed Image Data Pixels from processing Block620, which enters processing Block 640 along path 650, and ReferenceImage Data Pixels which enters processing Block 640 along path 615, toreplace each Mixed Image Data Pixel, with a variance exceeding the 1stThreshold, with its counterpart Reference Image Data Pixel, and generate1st Processed Mixed Image Data Pixels. 1st Processed Mixed Image DataPixels are the result of a first stage of ghost removal processing.

1st Processed Mixed Image Data Pixels are passed to processing Blocks685 and 695 along processing path 655. Processing Block 665 generatesthe Luma of 1st Processed Mixed Image Data Pixels, while processingBlock 660, from Reference Image Data Pixels which enters processingBlock 660 over path 615, generates the Luma of each Reference Image DataPixel. Processing Block 670 calculates the difference, on a pixel bypixel basis, between the Luma value of each 1st Processed Mixed ImageData Pixel and the Luma value of it's counterpart Reference Image DataPixel and provides these differences to processing Block 675. ProcessingBlock 675 determines the maximum value of these differences andprocessing Block 680 generates a 2nd Threshold based on this maximumvalue. Processing Block 695 receives this 2nd Threshold over path 685,along with Reference Image Data Pixels ever Path 615 and 1st ProcessedMixed Image Data Pixels over path 655 and replaces each 1st ProcessedMixed Image Data Pixel that exceeds this 2nd Threshold with itscorresponding Reference Image Data Pixel counterpart, and thus generatesenhanced ghost removed 2nd Processed Mixed Image Data on processing Path695. This 2nd Processed Mixed Image Data, the result of a 2nd stage ofghost removal processing, is used as input to a Tone Mapping processor,such as 235 of FIG. 2.

The Tone Mapping Process

In the preferred embodiment of the present invention, enhanced ghostremoved 2nd Processed Mixed Image Data on processing Path 695 of FIG. 6is 16 bits in bit-width and is comprised of 3 color components, a redcomponent, a green component and blue component. This RGB 16 bit data isto be displayed on Built-In Digital Camera Display 245 of FIG. 2, whichis an 8 bit display so the 2nd Processed Mixed Image Data needs to beconverted from 16 bit RGB data to 8 bit RGB data. The process ofconverting image data of one bit-width to that of a narrower bit width,such as from 16 bits to 8 bits, while maintaining the relative grayshade levels represented in the wider bit-width data in the resulting 8bit data, is referred to as “Tone Mapping”. There are many such ToneMapping processes that can be used by the present invention. Thepreferred embodiment of the present invention employs a unique tonemapping approach which was originally designed to map 12 bit wide imagedata to 8 bit wide image data. Therefore, the present invention approachfirst removes the 4 least significant bits of the 2nd Processed MixedImage Data, leaving 12 bit RGB image data. Three Look-Up fables (LUTs)are used to map the remaining 12 bit RGB data to the needed 8 bit RGBdata:

-   (a) A Normal Gain LUT,-   (b) A High Gain LUT; and-   (c) A Very High Gain LUT

The proper LUT to use in the 12 bit to 8 bit tone mapping process needsto be selected in order to correctly represent the image gray shadespresent in the 2nd Processed Mixed Image Data in 8 bit data form. Theselection criteria depends on the size of image area populated withpixels whose value, on average, is below a predefined pixel value, or“dark”, as compared to the rest of the image area. The lower the averagepixel value in the image dark area, the higher the gain of the LUTselected.

The process of LUT selection is as follows:

-   (1) Shift right the 12 bit RGB image by 4 bits. This results in an 8    bit image;-   (2) Generate the Luma component of the resulting 8 bit image;-   (3) Calculate the average value, Mn, of all pixels in the 8-bit    image whose Luma is less than a threshold for dark area, Td, A    digital value of 20 out of a maximum digital value of 255 (the    maximum 8 bit value) can be used for Td;-   (4) if the sum of all the pixels having Luma<Td is less than an area    threshold, P %, use the Normal Gain LUT, Otherwise:-   (5) Given Mn and predefined Thresholds pixel value thresholds T1    less than T2:-   (6) if Mn≦T1 use the Very High Gain LUT;-   (7) If Mn between T1 and T2 use the High Gain LUT-   (8) If Mn>T2 use the Normal Gain LUT

Td=20 out of 255, T1=5 out of 255 and T2=10 out of 255 are examples ofthe Thresholds that can be employed in the above Tone Mapping LUTselection process.

The tone mapping procedure employed by the present invention is designedto boost image regions of low illumination, while the 3 LUTs behave thesame for image regions of high illumination. It was found that thisprovides a good result when applied to the 2nd Processed Mixed ImageData of the present invention.

Having thus described several aspects of the preferred embodiment of thepresent invention, it is to be appreciated that various alterations,modifications, and improvements will readily occur to those skilled inthe art. Such alterations, modifications, and improvements are intendedto be part of this disclosure, and are intended to be within the spiritand scope of the invention. Accordingly, the foregoing description anddrawings are by way of example only.

What is claimed is:
 1. A method for generating a high dynamic range(HDR) image of a scene based on a series of images of a scene,comprising: ordering the series of images based at least on a differentexposure level for each image; selecting an image from among the seriesof images having an exposure level with a lowest value of the differentexposure levels as a reference image; blending together a first image ofthe ordered series and a second image of the ordered series to generatean intermediate image, wherein the second image of the ordered series iscaptured at a higher exposure level than the first image of the orderedseries, and wherein the second image is normalized to the exposure levelof the reference image; iteratively blending together the generatedintermediate image with a next unblended image of the series if theseries is composed of more than two images, wherein each blendedintermediate image is successively blended with another next unblendedimage of the series that is exposed at a next higher exposure level thana previously unblended image of the series, until each image of theseries is blended together in a last blended intermediate image, whereinthe other next unblended image is normalized to the exposure level ofthe reference image; and displaying a last generated intermediate imageas the HDR image of the scene.
 2. The method of claim 1, furthercomprising: capturing the series of images at different exposure levels;registering counterpart pixels for each image of the series to eachother; and deriving a normalized image exposure level for each image inthe series by using the exposure level of the image in the series thatis least exposed as a reference level.
 3. The method of claim 1, whereinblending further comprises: deriving a luma value for each pixel in thesecond image; using the luma value of a second image pixel as an indexinto a look-up table (LUT) to obtain a weighting value between thenumbers zero and unity; using the weighting value, the normalizedexposure level of the second image, and the second image pixel togenerate a processed second image pixel; selecting a first image pixelthat corresponds to the second image pixel; using the first image pixeland the result of subtracting the weighting value from unity, togenerate a processed first image pixel; adding the processed first imagepixel to the processed second image pixel to generate a blended imagepixel; and iterating the blending until each pixel of the second imageis blended with its corresponding pixel of the first image.
 4. Themethod of claim 3, wherein the obtained weighting value decreases as theluma value used as the index into the LUT increases.
 5. The method ofclaim 3, wherein a different LUT is used for each image for obtainingthe weighting value.
 6. The method of claim 1, further comprisingremoving a location shifted replication of an object in the scene thatappears in the generated intermediate image.
 7. A digital cameraoperative to capture a series of images of a scene, comprising: an imagecapture device configured to capture the series of image of the scene; amemory configured to store instructions; and a processor configured toexecute the instructions that, when executed, cause the processor toperform a process, comprising: ordering the series of images based atleast on a different exposure level for each image; selecting an imagefrom among the series of images having an exposure level with a lowestvalue of the different exposure levels as a reference image; blendingtogether a first image of the ordered series and a second image of theordered series to generate an intermediate image, wherein the secondimage of the ordered series is captured at a higher exposure level thanthe first image of the ordered series, and wherein the second image isnormalized to the exposure level of the reference image; iterativelyblending together the generated intermediate image with a next unblendedimage of the series if the series is composed of more than two images,wherein each blended intermediate image is successively blended withanother next unblended image of the series that is exposed at a nexthigher exposure level than a previously unblended image of the series,until each image of the series is blended together in a last blendedintermediate image, wherein the other next unblended image is normalizedto the exposure level of the reference image; and enabling a display ofa last generated intermediate image as a high dynamic range (HDR) imageof the scene.
 8. The digital camera of claim 7, further comprisinginstructions that, when executed, cause the processor to perform aprocess, comprising: capturing the series of images at differentexposure levels; registering counterpart pixels for each image of theseries to each other; and deriving a normalized image exposure level foreach image in the series by using the exposure level of the image in theseries that is least exposed as a reference level.
 9. The digital cameraof claim 7, wherein blending further comprises: deriving a luma valuefor each pixel in the second image; using the luma value of a secondimage pixel as an index into a look-up table (LUT) to obtain a weightingvalue between the numbers zero and unity; using the weighting value, thenormalized exposure level of the second image, and the second imagepixel to generate a processed second image pixel; selecting a firstimage pixel that corresponds to the second image pixel; using the firstimage pixel and the result of subtracting the weighting value fromunity, to generate a processed first image pixel; adding the processedfirst image pixel to the processed second image pixel to generate ablended image pixel; and iterating the blending until each pixel of thesecond image is blended with its corresponding pixel of the first image.10. The digital camera of claim 9, wherein the obtained weighting valuedecreases as the luma value used as the index into the LUT increases.11. The digital camera of claim 9, wherein a different LUT is used foreach image for obtaining the weighting value.
 12. The digital camera ofclaim 7, further comprising instructions that, when executed, cause theprocessor to perform a process, comprising removing a location shiftedreplication of an object in the scene that appears in the generatedintermediate image.
 13. A firmware stored on memory and includinginstructions for generating a high dynamic range (HDR) image of a scenebased on a series of images of a scene, wherein the execution of theinstructions causes a processor to: order the series of images based atleast on a different exposure level for each image; select an image fromamong the series of images having an exposure level with a lowest valueof the different exposure levels as a reference image; blend together afirst image of the ordered series and a second image of the orderedseries to generate an intermediate image, wherein the second image ofthe ordered series is captured at a higher exposure level than the firstimage of the ordered series, and wherein the second image is normalizedto the exposure level of the reference image; iteratively blend togetherthe generated intermediate image with a next unblended image of theseries if the series is composed of more than two images, wherein eachblended intermediate image is successively blended with another nextunblended image of the series that is exposed at a next higher exposurelevel than a previously unblended image of the series, until each imageof the series is blended together in a last blended intermediate imagewherein the other next unblended image is normalized to the exposurelevel of the reference image; and display a last generated intermediateimage as the HDR image of the scene.
 14. The firmware of claim 13,further comprising instructions that, when executed, cause the processorto: capture the series of images at different exposure levels; registercounterpart pixels for each image of the series to each other; andderive a normalized image exposure level for each image in the series byusing the exposure level of the image in the series that is leastexposed as a reference level.
 15. The firmware of claim 13, whereinblending further comprises instructions that, when executed, cause theprocessor to: derive a luma value for each pixel in the second image;use the luma value of a second image pixel as an index into a look-uptable (LUT) to obtain a weighting value between the numbers zero andunity; use the weighting value, the normalized exposure level of thesecond image, and the second image pixel to generate a processed secondimage pixel; select a first image pixel that corresponds to the secondimage pixel; use the first image pixel and the result of subtracting theweighting value from unity, to generate a processed first image pixel;add the processed first image pixel to the processed second image pixelto generate a blended image pixel; and iterate the blending until eachpixel of the second image is blended with its corresponding pixel of thefirst image.
 16. The firmware of claim 15, wherein the obtainedweighting value decreases as the luma value used as the index into theLUT increases.
 17. The firmware of claim 15, wherein a different LUT isused for each image for obtaining the weighting value.
 18. The firmwareof claim 13, further comprising instructions that, when executed, causethe processor to remove a location shifted replication of an object inthe scene that appears in the generated intermediate image.